spiceai-54-vortex-0.85.0 : cherry-picks - #101

…able

Traced through vortex-arrow's current Map handling instead of trusting the
merge-conflict shape: upstream 0.85.0 replaced the List<Struct<key,value>>
alias entirely with a native Map array/dtype (vortex-array/src/dtype/map.rs,
vortex-array/src/arrays/map/), and both the import and export paths already
recurse into map key/value fields for extension metadata the way Spice's
3e7fa40 did.

Ran the vortex-arrow map test suite on this branch (17 passing, including the
exact nested-extension-type scenario 3e7fa40 fixed) rather than asserting
this from the diff alone. Moves row 14 from "needs re-porting" to "dropped"
with that run as evidence.

…ys to timestamps instead of re-labelling (#75, #78, #93)

Re-ports two related fork patch families from spiceai-54 to 0.85.0 as fresh
implementations rather than cherry-picks, since neither applies: both touch
functions upstream restructured, and the second depends on the first already
being in place (resolve_timezone) to even compile against.

Fixed-offset timezone resolution (#75, #78): Arrow allows a timestamp's
timezone to be an IANA name or a fixed offset (+00:00), but jiff only
resolves IANA names, so any offset-form timezone aborted the process via
`.vortex_expect("unknown timezone")` -- in scalar validation, Display
rendering, and TemporalMetadata::to_jiff. Ports the fork's resolve_timezone
(new file, 38 passing tests carried over verbatim) and threads it through
all three sites, plus Scalar::try_extension_ref (a fallible counterpart to
the panicking extension_ref) through the three call sites that built
extension scalars infallibly on a read path (min/max aggregation, scalar_at,
the constant-folding reduce rule) and DateTimeParts::scalar_at.

Date -> timestamp conversion (#93, spiceai/spiceai#13624): Scalar::cast sent
any cast targeting an extension type through the target's storage type and
re-labelled the result, discarding what the source meant -- vortex.date and
vortex.timestamp both count from the epoch but in different units. A scan
falsifies `cast(col as timestamp) > lit` into `cast(max(col) as timestamp)
<= lit` and casts a file's statistic scalar to decide whether to read it at
all, so this silently pruned files whose rows would have matched. Ports
DateToTimestamp (new file, shared by both the array CastKernel and the
scalar cast so the two cannot disagree about which files/rows to keep) and
Timestamp::storage_range, replacing Jiff Span-based validation whose limits
are not a timestamp's at either end (span floor is one nanosecond above
i64::MIN, which a real timestamp[ns] column holds).

This supersedes a98a78c's array-only port on this branch: that port used
legacy_session() as a workaround for CastReduce's missing ExecutionCtx.
Discovered from an untracked later Spice commit that this fork's own answer
was to split into CastReduce (buffer-free restructuring) and a proper
CastKernel (which does get a real ctx) instead -- the same split this commit
uses, requiring an explicit `register_execute_parent_kernel(Cast.id(),
Extension, ...)` that has no upstream equivalent (Extension had no Cast
kernel at 0.85.0 at all, only the reduce path).

Verified: cargo test -p vortex-array -p vortex-datetime-parts --lib: 3428 +
72 passed, 0 failed. cargo check across vortex-array, vortex-arrow,
vortex-layout, vortex-file, vortex-io, vortex-utils, vortex-duckdb: clean,
zero warnings.

Not ported: the DuckDB TIMESTAMP_TZ dtype/scalar conversion's is_utc_timezone
gate (vortex-duckdb/src/convert/{dtype,scalar}.rs) -- vortex-duckdb is not a
crate Spice's Cargo.toml consumes, directly or transitively, so this is fork
hygiene rather than something Spice's re-pin needs. Left for a follow-up.

…e call site

vortex-file/src/writer.rs's array-context construction was rewritten to call
the lock-free, Vec-returning session.enabled_component_ids(...) once instead
of the SessionGuard-returning session.arrays() twice in one statement -- the
exact pattern the original patch existed to prevent. Swept every remaining
session.arrays() call site across the crates Spice consumes; none re-invoke
it within a live guard's scope. No code change needed; moves row 6 from
"needs re-porting" to "dropped" with that sweep as evidence.

…ude it in PRUNING_STATS

Re-ports a9ef29d from spiceai-54: this fork's own ledger only tracked the
vendored, out-of-scope half of the original patch (vortex-datafusion), never
noticing this core-crate half existed. Confirmed it doesn't exist at 0.85.0
independently of the aggregate-fn redesign that moved computation of the
stat itself into a new AggregateFnVTable module -- that redesign changes how
the value is computed per array type, not whether it survives PRUNING_STATS
or a dict take, so this is still needed on top of it.

Without PRUNING_STATS including it, the stat isn't guaranteed to be computed
at write time. Without take.rs propagating it (alongside Min/Max/IsConstant,
which already were), a Dict-encoded column's byte-size statistic silently
disappears after any take/filter, and the optimizer mis-sizes joins built
over what remains -- a planning regression, not a correctness one, but a
silent one either way.

Added two tests (none existed for propagate_take_stats before): one
confirming the new stat carries forward inexactly, one confirming
IsConstant's existing inexact-propagation path (added by the same original
patch) still fires correctly when an index is null and the exact path above
it doesn't apply.

Verified: cargo test -p vortex-array --lib: 3430 passed (up from 3428), 0
failed. cargo check across vortex-array, vortex-arrow, vortex-layout,
vortex-file, vortex-io, vortex-utils, vortex-datetime-parts: clean.

Read the full commit sequence rather than titles alone: the one non-perf
commit in the Sept-11 series (a panic fix in an extension list_contains
kernel) is itself deleted three commits later the same day, when the
extension-specific kernel it patched is replaced by generic-probe handling
-- matching 0.85.0, which never had that kernel. Not independently
reproduced against 0.85.0, so this is a reading, not a run, but the series
is not a live-bug row regardless: reframed from "needs its own audit" to "a
real but sizeable performance feature port against a since-rewritten
944-line file," deprioritized below the smaller items for that reason.

Also confirmed ab4f0b1 is moot (the file it touches doesn't exist at
0.85.0, the function it restores isn't referenced anywhere) and flagged
ccaa556's actual scope: it touches the central ExecutionCtx hot path, not
a leaf file, so it carries more blast radius than the other open items here.

…xpression

dafa096 (cherry-picked earlier on this branch as "Fix flat reader subrange
decode reuse") added a #[cfg(test)] module using APIs from spiceai-54's
2026-06-25 state: an undefined SESSION static (should be new_session()), an
ArrayContext passed without the .into() 0.85.0's write_stream signature
needs, and six call sites passing a raw Expression where LayoutReader's
filter_evaluation/projection_evaluation now require &BoundExpression
(upstream's "propagate bound expressions through layouts", vortex-data#9126).

This went undetected because the earlier verification pass ran `cargo check`
without --tests, so the test module's compile errors never surfaced --
surfaced only now while adding cargo check --tests coverage for a different
patch. Fixed by binding each expression once per branch with the same
`.bind(reader.dtype())` pattern the file's own flat_identity test already
used.

Verified: cargo test -p vortex-layout --lib layouts::flat::reader -- 9
passed, 0 failed. This is the first real test run of dafa096's actual
fix (segment-request coalescing across regressed query patterns), not
just a compile check.

…ts from host parallelism (#87)

Re-ports 4e2d626 from spiceai-54. Confirmed live at 0.85.0 before porting
(not assumed): ScanBuilder::with_concurrency's setting was multiplied by
get_available_parallelism() at two sites (scan_builder.rs, repeated_scan.rs),
so with_concurrency(1) still ran one split per worker thread rather than
serially -- a caller sizing a memory reservation against the configured
number under-reserves by the host's core count. MultiLayoutDataSource's
deferred-file-open concurrency had the same defect one level worse: it used
get_available_parallelism() as the value directly, so file-open width scaled
with the node under a cgroup CPU quota narrower than the machine, the same
class of bug crates/cpu-budget exists to prevent elsewhere.

Introduces SplitConcurrency (PerWorker/Absolute) with one effective() that
resolves either into a width, so a caller reading the setting and the scan
honoring it cannot disagree about what it meant. with_absolute_concurrency
is new; with_concurrency keeps its existing per-worker meaning.
MultiLayoutDataSource's deferred-open default is now the fixed
DEFAULT_CONCURRENCY (8), not the host's core count.

The bound only holds if split construction is lazy under it: constructing a
split task registers its reads with the I/O system, so building every task
up-front (the previous LazyScanState::Preparing / RepeatedScan::execute
path) let with_absolute_concurrency(1) still register every split's reads
before the first one was polled. RepeatedScan::execute now delegates to a
new split_tasks() lazy iterator that RepeatedScan::execute_stream buffers by
the configured concurrency, and ScanBuilder::into_stream only runs `prepare`
on the CPU pool before streaming through the same lazy path rather than
collecting every task first. execute()/build() are unchanged in meaning
(unbounded by the setting) and now say so in their docs.

Ported the two regression tests from the original patch (previously none
existed for this): both count registered splits via a new SplitProbe/
ProbeLayoutReader test harness in scan/test.rs and assert the high-water
mark equals the configured concurrency, covering absolute concurrency 1 and
3, ordered and unordered, at both the RepeatedScan and ScanBuilder layers.

Verified: cargo test -p vortex-layout --lib: 255 passed (up from 249), 0
failed -- including all 6 new/changed test cases. cargo test -p vortex-file
--lib: 139 passed, 0 failed (downstream consumer of the scan path). cargo
check --tests across vortex-array, vortex-arrow, vortex-layout, vortex-file,
vortex-io, vortex-utils, vortex-datetime-parts: clean, zero warnings.

Not ported: the vortex-file import-reordering hunks in this original commit
(driver.rs, request.rs, segments/source.rs) are pure rustfmt churn already
reflected in 0.85.0's own ordering; nothing to apply.

vortex-layout/src/scan/split_by.rs at 0.85.0 already carries 26b274c's
subdivide_large_spans function and all 5 of its unit tests, byte-for-byte
identical apart from unrelated test-helper API drift -- landed via upstream
commit 9d3aafb, which the row's own "Possibly upstream" guess had already
flagged as a candidate origin. vortex-file's end-to-end test is present too.
Ran both test suites rather than trusting the diff match alone. Moves the
row from "needs re-porting" to "dropped".